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ABSTRACT 



An image search and retrieval system and method of opera- 
tion within such a system are disclosed. The image search 
and retrieval system includes a search engine coupled to an 
image analyzer, an input system, an output system and an 
image database. The image search and retrieval system 
includes one or more feature modules which define particu- 
lar regions of an image and particular measurements to make 
on pixels within the defined image region as well as the 
measurements to make on neighboring pixels in neighboring 
image regions for a given application and communicate this 
application specific information to the image analyzer 
thereby allowing rapid comparisons of images. 

26 Claims, 30 Drawing Sheets 
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INFORMATION SEARCH AND RETRIEVAL Still other systems classify images or video frames by 

SYSTEM using image statistics including color and texture. The 

difficulty with these systems is that for a given query image, 

CROSS-REFERENCE TO RELATED even though the images located by the system may have the 

APPLICATIONS 5 samc color, textural, or other statistical properties as the 

This application claims priority under 35 U.S.C. §119(e) example image, the images may not be part of the same class 

from application No. 60/063,623 filed Oct. 27, 1997. 35 tbc q ucrv ima S c * n * 1 ucr y ima S e belong? to the 

class of images identified as human faces, then systems 

STATEMENTS REGARDING FEDERALLY which classify images or video frames based image statistics 

SPONSORED RESEARCH io including color and texture may return images which fall 

THis invention was made with government support under ™ lhin me ***** and texture ^ slatistics but 

Grant No. NOO014-95-1-O600 awarded by the U.S. Navy. which m not human faces " 

The government has certain right in the invention. " w ° uld ^us be desirable to provide a system and 

technique which may be used in a general image search and 

BACKGROUND OF THE INVENTION 15 retrieval system and which allows searching of a plurality of 

This invention relates to information search and retrieval differei11 'H** °f images including but not limited to human 

systems and more particularly to search and retrieval sys- or f mmal faces > fabnc Patterns symbok, logos, art gallery 

terns which utilize in whole or in part image processing and mus f m management, architecniral image and design 

techni es interior design, remote sensing and management ot earth 

A . * . 4 . , . . 20 resources, geographic information systems, scientific data- 

As is know in fre art, a digital .mage is an image which ^ m ment> wcather f orccasting> retailing , fabric and 

may be represented as an array of prxeU with each of the fashion trademark and copyright databasc 

p«els represented by a rhgita word. Often the array _« emeaU law enforcemellt investigation, 

provided as a two-dunensiona^ I array of pixels. Wfch the £ living, communication systems and inspection 

mcreasem the number of avajlable digital p^tures, the need 2S ^ circuit inspection S y Stems . h would be 

has arisen for more complete and efficient annotation desirable t0 have ^ system be of 

(attaching identifying labels to images) and indexing £ utomaticall learnm whicb factors ^ most importanl m 

(accessirig speafic images from the database) systems. Digi- aealdbSag for a or for a particular type of 
tal image/video database annotation and indexing services 

provide users, such as advertisers, news agencies and maga- 30 una S e - 

zine publishers with the ability to browse through, via SUMMARY OF THE INVENTION 
queries to an image search system, and retrieve images or \ Q accordance with the present invention, image process- 
video segments from such databases. fag system includes a search engine coupled to an image 

As is also known, a content based image retrieval system analyzer. The image analyzer and search engine are coupled 

is an image retrieval system which classifies, detects and 35 to one or more feature modules which provide the informa- 

retrieves images from digital libraries by utilizing directiy tion necessary to describe how to optimize the image 

the content of the image. Content based image processing analyzer for a particular application. With this particular 

systems refer to systems which process information in an arrangement, an image processing system which can rapidly 

image by classifying or otherwise identifying subject matter match a primary image to a target iomage is provided. Each 

within the image. Such systems may be used in a variety of 40 feature module defines particular regions of an image and 

applications including, but not limited to, art gallery and particular measurements to make on pixels within the 

museum management, architectural image and design, inte- defined image region as well as the measurements to make 

rior design, remote sensing and management of earth on neighboring pixels in neighboring image regions for a 

resources, geographic information systems, scientific data- given application. The feature modules thus specify param- 

base management, weather forecasting, retailing, fabric and 45 cters and characteristics which are important in a particular 

fashion design, trademark and copyright database image match/search routine. The phig-in-modules commu- 

management, law enforcement and criminal investigation nicate this application specific information to image ana- 

and picture archiving, communication systems and inspec- lyzer. The information specified by a particular feature 

tion systems including circuit inspection systems. module will vary greatly depending upon the particular 

Conventional content based image/video retrieval sys- 50 application. By using the feature modules, generic search 

terns utilize images or video frames which have been engines and image analyzers can be used. Thus the system 

supplemented with text corresponding to explanatory notes can be rapidly adapted to operate in applications as widely 

or key words associated with the images. A user retrieves varying as inspection of printed circuit boards or integrated 

desired images from an image database, for example, by circuits to searching for trademark images. In each 

submitting textual queries to the system using one or a 55 application, the particular parameters and characteristics 

combination of these key words. One problem with such which are important are provided by the feature module, 

systems is that they rely on restricted predefined textual It should thus be noted that the techniques of the present 

annotations rather than on the content of the still or video invention have applicability to a wide variety of different 

images in the database. types of image processing applications. For example, the 

Still other systems attempt to retrieve images based on a 60 techniques may be used in biometric applications and sys- 

specified shape. For example, to find images of a fish, such terns to identify and or verify the identity of a particular 
systems would be provided with a specification of a shape of person or thing, inspection systems including inspection and 

a fish. This specification would then be used to find images test of printed circuit boards (including without limitation 

of a fish in the database. One problem with this approach, any type of circuit board or module or integrated circuit) in 

however, is that fish do not have a standard shape and thus 65 all stages of manufacture from raw boards to boards which 

the shape specification is limited to classifying or identifying are fully assembled and/or fully loaded with components 

fish having the same or a very similar shape. and sub-assemblies (including hybrids), including solder 
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joint inspection, post paste inspection and post placement 
inspection, inspection and test of semiconductor chips in all 
stages of manufacture, from wafers to finished chips, image 
or video classification systems used to search image or video 
archives for a particular type of image or a particular type of 
video clip, and medical image processing applications to 
identify particular characteristics in an image, such as a 
tumor. Thus the phrase "image processing application" or 
more simply "application" as used hereinbelow refers to a 
wide variety of uses. 

In accordance with a further aspect of the present 
invention, a process for comparing two images includes the 
steps of (a) aligning a target image and a selected image, (b) 
dividing the selected image into a plurality of image regions, 
(c) collapsing properties in predetermined image regions, (d) 
selecting a primary image region, (e) selecting a target 
image region, and (f) comparing one or more properties of 
the selected primary image region to corresponding one or 
more properties in the target image region. With this par- 
ticular arrangement, a technique for comparing rapidly two 
image is provided. By selecting predetermined features and 
image regions to compare between the two images, the 
amount of time require to process the images is reduced. By 
combining or collapsing features in a selected regions of an 
image for compact representation, a comparison of a rela- 
tively large amount of information can be accomplished 
rapidly. 

In accordance with a still further aspect of the present 
invention, a method of manufacturing a printed circuit board 
includes the steps of (a) performing a manufacturing opera- 
tion on a printed circuit board; and (b) inspecting the result 
of the manufacturing operation by comparing an image of 
the actual operation being performed to a target image of the 
manufacturing operation. With this particular arrangement, 
the efficiency of manufacturing a printed circuit board is 
increases while reducing the cost of manufacturing the 
printed circuit board The manufacturing can correspond to 
any one or more steps in the printed circuit board (PCB) 
manufacturing process. For example, when the manufactur- 
ing process corresponds to a solder manufacturing process, 
then the inspection technique may be used before and/or 
after the post-paste, post-placement and post-reflow opera- 
tions. For example, the manufacturing process can include 
the inspection before and or after solder application, com- 
ponent placement, solder reflow, solder joint inspection or 
any other manufacturing step. By inspecting after predeter- 
mined steps in the manufacturing process, it is possible to 
detect early defects in the manufacturing process which can 
be corrected prior to continuing the manufacturing process. 
By detecting defects early in the manufacturing process, the 
expense and time associated with manufacturing a PCB 
which cannot pass a final inspection test is provided. 

In one embodiment, a camera captures an image of a 
printed circuit board and compares that image to a target 
image. The target image corresponds to an image of a PCB 
which has been properly manufactured in during that par- 
ticular portion of the manufacturing process. For example, if 
the manufacturing step of interest is a solder paste applica- 
tion step, then the captured corresponds to an image of a 
printed circuit board (or portion thereof) to which solder 
paste has been applied. The target image corresponds to an 
image of a PCB (or portion thereof) having solder paste 
properly applied thereto. Predetermined features of the two 
images are compared and based on the results of the 
comparison, decision is rendered as to whether the solder- 
paste was properly applied to PCB currently undergoing the 
manufacturing process. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
The foregoing features of this invention, as well as the 
invention itself, may be more fully understood from the 
following detailed description of the drawings in which: 
5 FIG. 1 is a block diagram of an image search and retrieval 
system; 

FIG. lAis a diagram of a query image and a plurality of 
images resultant from submitting the query image to the 
image search and retrieval system of FIG. 1; 

to FIG. 2 is a block diagram of an image processing system 
including a plurality of plug-in modules; 

FIGS. 3-3B are a series of flow diagrams representing the 
processing performed by the image processing system of 
FIGS. 1 and 2 to compare two image; 

15 FIG. 3C is a plot of attributes from a query image and a 
target image; 

FIG. 3D is a series of images illustrating an image after 
a divide and collapse operations; 

FIGS. 3E and 3F illustrate deformation of a region to 
20 match a target image; 

FIGS. 4~4B are a diagrammatic view the alignment of a 
primary image and a target image; 

FIG. 5 is a flow diagram showing the processing per- 
formed to generate search results; 
25 FIGS. 6 and 6A are a series of images showing the steps 
involved in collapsing regions of an input image to generate 
a collapsed image to be provided to an image search engine; 

FIGS. 6B and 6C are a series of images illustrating the 
comparison of a collapsed core image to a collapsed data- 
30 base image; 

FIGS. 7 and 7A are a series diagrammatical views show- 
ing a ripple feature; 

FIG. 7B is a flow diagram explaining the steps to imple- 
ment a ripple feature; 
35 FIG. 8 is an illustration of the selection of one or more 
resulting images from an image search; 

FIGS. 9-9F are a series of images which illustrate how a 
series of images which illustrate how a query image may be 
obtained and how to emphasize one or more portions of a 
40 query image; 

FIG. 10 is a flow diagram illustrating the steps to process 
a target using multiple subsections of a primary image; 

FIGS. 10A-10C are a series of images illustrating the 
placement of a selected image region on a target image; 
45 FIG. 11 is a flow diagram showing the steps to compute 
a score and provide an output to a user; 

FIG. 11A is a flow diagram showing the steps to compute 
an aggregate score; 

FIG. 11B is a flow diagram showing the steps to compute 
50 a score using the intersection method; 

FIG. 12 is a flow diagram showing the steps to perform 
sorting by text and then by visual similarity; 

FIGS. 12A-12D are a series of views illustrating textual 
annotations used to locate images and then sorting the 
55 located images by visual similarity; 

FIG. 13 is a diagrammatical side view of a circuit com- 
ponent and a circuit lead connected to a circuit line; 

FIG. 14 is a flow diagram showing the steps to perform 
image processing in a circuit inspection system; and 
60 FIG. 15 is a diagram showing the steps to manufacture a 
printed circuit board. 

DETAILED DESCRIPTION OF THE 
INVENTION 

65 Before describing an image search and retrieval system 
and the techniques associate therewith, some introductory 
concepts and terminology are explained. 
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An analog or continuous parameter image such as a still 
photograph may be represented as a matrix of digital values 
and stored in a storage device of a computer or other digital 
processing device. Thus, as described herein, the matrix of 
digital data values are generally referred to as a "digital 
image" or more simply an "image" and may be stored in a 
digital data storage device, such as a memory for example, 
as an array of numbers representing the spatial distribution 
of energy at different wavelengths in a scene. 

Similarly, an image sequence such as a view of a moving 
roller-coaster for example, may be converted to a digital 
video signal as is generally known. The digital video signal 
is provided from a sequence of discrete digital images or 
frames. Each frame may be represented as a matrix of digital 
data values which may be stored in a storage device of a 
computer or other digital processing device. Thus in the case 
of video signals, as described herein, a matrix of digital data 
values are generally referred to as an "image frame" or more 
simply an "image" or a "frame." Each of the images in the 
digital video signal may be stored in a digital data storage 
device, such as a memory for example, as an array of 
numbers representing the spatial distribution of energy at 
different wavelengths in a scene in a manner similar to the 
manner in which an image of a still photograph is stored. 

Whether provided from a still photograph or a video 
sequence, each of the numbers in the array correspond to a 
digital word (e.g. an eight-bit binary value) typically 
referred to as a "picture element" or a "pixel" or as "image 
data." The image may be divided into a two dimensional 
array of pixels with each of the pixels represented by a 
digital word. 

Reference is sometimes made herein to color images with 
only a luminance component. Such images are known as 
gray scale images. Thus, a pixel represents a single sample 
which is located at specific spatial coordinates in the image. 
It should be noted that the techniques described herein may 
be applied equally well to either grey scale images or color 
images. 

In the case of a gray scale image, the value of each digital 1 ^ 
word corresponds to the intensity of the pixel and thus the 
image at that particular pixel location. 

In the case of a color ima ge, reference is sometimes made 
heretrT to ea ch pixel bein g represented by a predetermined 
nUmber*of b~itsje.g. eight bits) which represent the color re d 
(R'bits), a predetermines number of bits (e.g. eight bits) 
which represent the color green (G Sitsj ana a preoetermined 



ffbm~mT"RGB3^uesT : Thus— in-an-eipt3iCcobr_RGB 
representedon,.a^pixel-may-te-represe nted by a twenty-four 
bitdi gital word , 

It is of course possible to use greater or fewer than eight 
bits for each of the RGB values. It is also possible to 
represent color pixels using other color schemes such as a 
hue, saturation, brightness (HSB) scheme or a cyan, 
magenta, yellow, black (CMYK) scheme. It should thus be 
noted that the techniques described herein are applicable to 
a plurality of color schemes including but not limited to the 
above mentioned RGB, HSB, CMYK schemes as well as the 
Luminosity and color axes a & b (Lab) YUV color difference 
color coordinate system, tbe Karhunen-Loeve color coordi 
nate system, the retinal cone color coordinate system and the 
X, Y, Z scheme. 

Reference is also sometimes made herein to an image a: 
a two-dimensional pixel array. An example of an array size' 



is si2e 512x512. One of ordinary skill in the art will of 
course recognize that the techniques described herein are 
applicable to various sizes and shapes of pixel arrays includ- 
ing irregularly shaped pixel arrays. 

A "scene" is an image or a single representative frame of 
video in which the contents and the associated relationships 
within the image can be assigned a semantic meaning. A still 
image may be represented, for example, as a pixel array 
having 512 rows and 512 columns. An "object" is an 
identifiable entity in a scene in a still image or a moving or 
non-moving entity in a video image. For example, a scene 
may correspond to an entire image while a boat might 
correspond to an object in the scene. Thus, a scene typically 
includes many objects and image regions while an object 
corresponds to a single entity within a scene. 

An "image region" or more simply a "region" is a portion 
of an image. For example, if an image is provided as a 32x32 
pixel array, a region may correspond to a 4x4 portion of the 
32x32 pixel array. 

Before describing the processing to be performed on 
images, it should be appreciated that, in an effort to promote 
clarity, reference is sometimes made herein to one or more 
"features" or "information" in "blocks" or "regions" of an 
image. It should be understood that the features can corre- 
spond to any or any particular characteristic of the block 
including its relationship to other blocks within the same or 
a different image. Also, such image blocks or regions should 
be understood as not being limited to any particular type, 
size or shape of a portion of an image (i.e. the block need not 
have a square or a rectangular shape). It should also be 
understood that the image need not be any particular type of 
image. 

Similarly, reference is also sometimes made herein to 
comparison of the "features" or "information" in one of 
more blocks to "features" or "information" in one or more 
other blocks. The other blocks may be from the same or a 
different image than the first blocks. Also the processing of 
the blocks need not be for any specific type of image 
processing application. Rather the processing of the blocks 
applies in a wide variety of image processing applications. 
^ Accordingly, those of ordinary skill in the art will appre- 
ciate that the description and processing taking place on 
"blocks" and "regions" could equally be taking place on 
portions of an image having a square, rectangular, triangular, 
circular, or elliptical shape of any size. Likewise, the par- 
ticular field in which the image processing systems and 
techniques of the present invention may be used includes but 
is not limited to, biometric applications and systems to 
identify and or verify the identity of a particular person or 
thing, inspection systems including inspection and test of 
printed circuit boards (including without limitation any type 
of circuit board or module or integrated circuit) in all stages 
of manufacture from raw boards to boards which are fully 
assembled and/or ttully loaded with components and sub- 
assemblies (including hybrids), including solder joint 
inspection, post paste inspection and post placement 
inspection, inspection and test of semiconductor chips in all 
stages of manufacture, from wafers to finished chips, image 
or video classification systems used to search image or video 
archives for a particular type of image or a particular type of 
video clip, and medical image processing applications to 
identify particular characteristics in an image, such as a 
tumor. 

Referring now to FIGS. 1 and 1A, an image search and 
retrieval system 10 includes an image processing system 12 
having an input system 14 and output system 16 coupled 
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thereto. Input System 14 is here shown to include a graphical 
user interface (GUI) 14a, a facsimile system 146, a camera 
system 14c which may include for example a digital camera 
or a camera connected to a frame grabber, a scanner 144 a 
network connection 14e, a video system 14N which, may 
include a video camera for example, and any other means 
known to those of ordinary skill in the art for capturing or 
otherwise obtaining images and transmitting or otherwise 
making image data available to an image processing system. 

For example, Network Connection 14e allows input sys- 
tem 14 to receive image data from a global information 
network (e.g., an internet) or on intranet or any other type of 
local or global network. Also coupled to image processing 
system 12 is an image storage device 18 which may for 
example be provided as image one or more databases having 
stored therein a plurality of images 20a-20N generally 
denoted 20. The images 20 may be provided as still images 
or alternatively the images may correspond to selected 
frames of a video signal which are treated as still images. 

Image processing system 12 also includes an image 
retrieval to retrieve images from the storage device 18. 
Alternatively, images may be provided to image processing 
system 12 via the graphical user interface 14a using one of 
a number of commercially available drawing packages such 
as Microsoft Paint or any similar package. Alternatively still, 
a camera such as camera 14c or other image capture device 
may be used to feed images to the processing system 12. 
Thus system 12 can receive real time or "live" camera 
images instead of retrieving images from a database or other 
storage device. Alternatively still, images may be fed to the 
imaging processing system 12 via the facsimile system 14b, 
the scanner 14d or the like. Regardless of how images are 
fed to image processing system 12, image processing system 
12 receives the images and processes them in accordance 
with techniques to be described below in conjunction with 
FIGS. 2^8B. 

In general overview and as shown in FIG. 1A, a query 
image 20 is provided to image processing system 12 via 
input system 14. The image processing system 12 compares 
the query image 20 to each of the images in the storage 
device 18 and computes a quantitative value or "score" 
indicative of the closeness of the match to the query image. 
In the case where the storage device is provided as an image 
database, the processing system 12 accesses the database via 
a database query. 

Image processing system 12 provides the results of the 
query to output system 16 in a pictorial format as a plurality 
of images 22a-22/t. Each of the result images 22a-22h has 
a quantitative value 24a-24h associated therewith. The 
quantitative values 24a-24h indicates the closeness of the 
match between the query image and the image stored in the 
storage device 18. A quantitative value 24 closest to a value 
of zero indicates the closest match while higher values 
indicate a match which is not as close. The quantitative 
values allow the image processing system 12 to order the 
result images. The relative differences between the quanti- 
tative values associated with each image can be used to 
cluster groups of images based upon the closeness of the 
match to the query image. For instance there is a relatively 
large difference between the scores associated with images 
22b and 22c. This provides an indication that images 22a 
and 22b are closer in terms of visual similarity to the query 
image, 20, than the images 22c, 22d and beyond. The 
difference in scores between 22/ and 22g is much smaller 
than the difference in scores between 22b and 22c. This 
indicates that images 22/ and 22g in relation to the query 
image, 20, are very similar. Thus, it should be appreciated 



3,426 Bl 

8 

that it is the relative values of the scores rather than the 
absolute scores can be used to order the images found as a 
result of the search. 

Referring now to FIG. 2, image processing system 12 

5 includes an image processor 24, a search engine 36, an 
image analyzer 38, and a storage device interface 39. Image 
processor 24 interfaces with one of a plurality of plug-in 
modules 30a-30N generally denoted 30. 

The function of the plug-in-mod ules 30 is to provide the 

10 information necessary to describe how to optimize the image 
analyzer 38 for a particular application. Each of the plug- 
in- modules 30 define the particular regions of an image and 
particular measurements to make on pixels within that image 
region as well as the measurements to make on neighboring 

15 pixels in neighboring image regions for a given application. 
For example, Module 30a specifies parameters, constraints 
and characteristics to be used when performing trademark 
image searches. Module 30b specifies parameters, con- 
straints and characteristics to be used when performing 

20 searches upon images of faces. Module* 30k specifies 
parameters, constraints and characteristics to be used when 
performing searches of scenes (e.g., images of waterfalls, 
fields, etc.). Module 301 specifies parameters, constraints 
and characteristics to be used when performing searches for 

25 fabrics. Module 30/ specifies parameters, constraints and 
characteristics to be used when performing inspection of 
printed circuit boards. Module 30N specifies parameters, 
constraints and characteristics to be used when performing 
searches on a stream of video images. The plug-in-modules 

30 30 thus specify parameters and characteristics which are 
important in a particular image match/search routine. The 
plug-in-modules communicate this application specific 
information to image analyzer 38. 

f pl The plug-in-modules 30 may be implemented in a variety " 

35 of ways. The modules 30 themselves may implement a 
technique that takes as in put two or more image regions , 
compares them to each other, and returns scores or now 
similar they are to each other. The plug-in-modules 30 may 
contain a list of image attributes that _a re, im portant fbr 

40 computin g image similarity for a particular application, such 
a s^color, luminance, and texture 'ThlTpluR^in modules may 
also suggest how to preprocess images before the images are 
sent from storage device interface 39 to image analyzer 38. 
For example, if the query image is a trademark image and 

45 it is desirable to search the storage device 18 (FIG. 1) for 
trademark images, then the trademark plug-in -module 30a 
describes that orientation information is important. It also 
tells the image analyzer to ignore color information and to 
treat black figures on a white background and white figures 

50 on a black background in the same manner. The trademark 
plug-in-module may also describe how to compute orienta- 
tion information and how to compact this information fbr 
more efficient processing. These techniques are more fully 
described in FIGS. 6-6C. 

ss^pOn the other hand, if the application is face recognition 
'and the query image contains a human face, then plug-in 
module 30b, which corresponds to a face optimizer, would 
be used. The face plug-in-module 306 contains information 
describing that global configuration is important and relative 

60 luminance or relative color information should be used in 
the matching process to be robust against illumination 
changes. Although, not necessary, plug-in-module 30b may 
implement a technique that describes how to delineate the 
boundaries of a face in an image. This technique may be 

65 used to preprocess the query hand target images prior to 
analysis. Such a technique may be used to align the query 
and the target image (as shown in step 40 of FIG. 3). 
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aS? The scene plug-in-module 3<Mr emphasizes color and local 
^and global structure as important attributes when comparing 
two scenes. The fabric optimizer 30* uti lizes a weighted 
combination of oolor. texture, local andg l obal structure in a 



simi iantv met jjcLwhich ta kes as input two or more images of 
fahne jamples. A video sequence plug-in-module can also 
be used. In the case of the video optimizer 30N, it must 
describe how to match one still image to another image and 
also how to take into account that these images are part of 
a time sequence. For instance, it may be important to track 
objects over time. It may also be i mportant to ca lc ulate the 
relative position, color, or other attribute from a trame at 



10 



time tl and a frame at time t2. These calculations can E e 
made Independently for two or more image sequences._The 



20 



processed jjna^csequcDccs may then be compared 

It is not necessary to have a predefined specialized 
plug-in-module in order for image analyzer 38 to compare 
two images. The image analyzer may use a default plug-in- 
module 30k, which contains generic biases to compare two 
or more images of unknown content. For instance, the 
default plug-in-module 30k utilize color, luminance, and* 
local relative structure. If feedback is provided to the image 
processor 24 and the plug-in module, the default plug-in- 
module 30x can generate a list of associations to important 
image attributes. Me asuring and comparing these attribu tes 
will ther efore take precedence over measurin g and compa r- 
ing otKefattributes" In the learning step the default plug-in- 
mo"3u!e30jt can also identify significant internal parameters 
and ranges for their values. This learning or refining proce- 
dure can be applied to the default plug-in-module 3(Mt or any 
of the application specific plug-in-modules 30a-30to. The 
learning procedure can be used to generate a set of 
parameters, characteristics and constraints for a new plug-in 
module or to train an existing plug-in-module to utilize a set 
of parameters, characteristics and constraints for a newj^ 5 
application. 

Many of the applications described are image search 
applications. The basic premise is that one or more images 



30 



generate computer software to perform the processing 
required of the particular apparatus. It should be noted that 
many routine program elements, such as initialization of 
loops and variables and the use of temporary variables are 
not shown. 

Some of the processing blocks can represent an empirical 
or manual procedure or a database function while others can 
represent computer software instructions or groups of 
instructions. Thus, some of the steps described in the flow 
diagram may be implemented via computer software while 
others may be implemented in a different manner e.g. 
manually, via an empirical procedure, or via a combination 
of manual and empirical procedures. 

It will be appreciated by those of ordinary skill in the art 
that unless otherwise indicated herein, the particular 
sequence of steps described is illustrative only and can be 
varied without departing from the spirit of the invention. 

Referring now to FIGS. 5-3B, processing begins in step 
40 where a selected or primary image is aligned with a target 
image. The primary image is the image which is to be 
matched to the target image. In some embodiments, the 
target image has been previously stored in a database (such 
as database 18 in FIG. 1) and is retrieved therefrom while in 
other embodiments the target image is not retrieved from a 
database. 

Either after or before the alignment step, the primary and 
target images are each divided or segmented into a plurality 
of sub regions (or more simply regions) or blocks as shown 
in step 42. It should be noted that divide step 42 may be 
performed as a so-called intelligent divide. An intelligent 
division of the image is a procedure in which regions of 
pixels that have similar attributes are formed into groups. 
One example is to group regions according to color. For a 
field scene, which has a blue sky and a green field, the 
intelligent divide would partition the image into the blue and 
green parts. Another example of an intelligent divide is to 
group pixels that belong to the same object. For instance, in 
an image of a printed circuit board, the intelligent divide 
would group image regions that belong to the same corn- 



are provided to the image processor 24 as positive or 

negative examples or samples. Together these examples 40 ponent. In an alternate embodiment, the intelligent divide 



define a semantic class of images. The task of the image 
processor 24 is to search one or more images (which may be 
stored in one or more databases) to find other images in this 
class. Search engine 38 can be optimized to perform 
searches rapidly. In its most generic form, the search engine 
38 is_used to ret rieve images fr om a storage device and to 
channel those images sequentially or in batches to the image 
analyzer. Images from the search engine 36, which are 
identified by the image analyzer 38 or by a user may be used 



45 



can be performed on the target images in place of or in 
addition to the primary image, 
/y^ In ste p 44. the properties in predetermined ones of the *~\ 
subregion s^or blocks of the primary image arc combined or 
collapsed" to provide a relatively compact representation of 



me"p»rimaryrOne particular tecHmque to collapse the data in 
me~subTegions is to get the average value of a part icular 
paramet er _over J he whole subregion. For instance, this 
collaps ing stepcm53^eTo^fina > TBe average luminance of th e 
to refine a search or to perform a new search. This is further 50 s ub "reg ion. Another particular technique for combining the 



explained in conjunction with FIGS. 7-12 below. 

FIGS. 3-3B, 5, 7B, 10, 11D, HE, 12 and 14 are a series 
of flow diagrams showing the processing performed by a 
processing apparatus which may, for example, be provided 
as part of an image processing system such as that shown in 
FIGS. 1 and 2 to allow processing of various types of images 
in a variety of different applications. The rectangular ele- 
ments in the flow diagrams (typified by element 40 in FIG. 
3) are herein denoted "processing blocks" and represent 
computer software instructions or groups of instructions. 

Alternatively, the processing blocks represent steps per- 
formed by functionally equivalent circuits such as a digital 
signal processor circuit or an application specific integrated 
circuit (ASIC). The flow diagram does not depict the syntax 
of any particular programming language. Rather, the flow 
diagram illustrates the functional information one of ordi- 
nary skill in the art requires to fabricate circuits, or to 
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65 



block properties is described hereinbelow in conjunction 
with FIGS. 6-6C. It should be noted that step 44 is an 
optional step and in some embodiments it may not be 
necessary or desirable to perform the collapsing step. In 
other embodiments, however, to increase the speed in 
matching the primary and target images, it may be desirable 
or necessary to perform the collapsing step. It should be 
noted that divide step 42 may be performed either before or 
after the collapsing step 44. w 
Processing then flows to step 46 in which a region or 
block within the primary image (referred to herein as a 
primary image region) is selected. Each primary image 
region has associated therewith a set of properties or char- 
acteristics or features. In some embodiments, the primary 
image region and regions within a predetermined distance of 
the primary image region (referred to herein as neighbor 
regions) are used in the matching process. 
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In the case where neighbor regions are used, the proper- then the score can be augmented by adding in difference in 

ties of both the primary image region and the neighbor luminance and position of the corresponding primary and 

regions are used in the matching process. In one target region neighbors. Often, the differences in the neigb- 

embodiment, a radius Rl is selected to determine which bors characteristics are weighed less than the difference in 

neighbor regions around the primary image region to include 5 the regions themselves. For instance, the weights of the 

for further processing. For example, a radius of zero (i.e. neighbors can be calculated by a gaussian function centered 

R1=0) indicates that no neighbor regions have been on the location ofthe primary image region. Other weighting 

included, where a radius greater than zero (e.g. Rl=l) would functions may of course also be used, 
indicate that at least some neighbor regions are to be If ^ R1 and R2 are grea t er than zero, then relative 

inc uded The particular number of neighbor regions to 10 measures ^ made between mc ri ^ e region 

include should be selected in accordance with a variety of and jts a ^ Ama or ^ neighbors to each other. For 

factors including but not limited to the particular ^ one can u , e whethet the nei ^ bors have 
application, the processing speed and the relative impor- r ^ Qr j luminlnce to me ri ^ 

tance of particular structure in the matching process. Thus, re ^ oQ ^ same ^ be made be , ween lh X 

one reason to include neighbors m the ^matching process is 15 rf ioQ and ils ^ meKD( l 

if the structure mcluded in the neighbor is an important betw£en ^ primary j,^ region and the Urget image! 

considerauon in the matching process. For instance, when K ■ ^ te ef tf order of , hese relative ^0^,^ 

companng facial images, the structure of the face is very fc m ^ ; ion Differences m relative I 

important. In all cases there should be regions corresponding measures ^ weighted and addcd into the total score. Y 
to the eyes, nose, mouth, cheeks, etc . . . These must be 2n . . 

configured in thecorrect spatial organization (e.g. the cheeks 2 ° A Processing then flows to dec*™ block 54 where it is 

should be on either side of the note). The magnitude of the determined whether it is necessary to compare *e properties 

radius Rl indicates the level at which the structure is ofmepruriary unage region tome propemes of a next target 

important. A radius Rl having a relatively small value P nmar y J™* «ffon. U it is necessary to compare the 

indicates that local structure is important. As the value of „ P nmarv . ™& le *™ *° a *?* target image region, then 

radius Rl increases to the limit of thVheight and width of the * P™f»°g floWS l° ! P 1 ' \ 2? m3ge K& °* 

image, then global structure is emphasized. * t fP 58 ' ^ hcr f th ? P™ 31 * m f c 

t „, ... . _j region is aligned with the next target primary region for 

It should be noted that not all the neighbor regions around flf £ m p,^^ thet! 1 returns to step 48. 

the pnmary unage region are importanL For instance, when ^ 54 56 and 58 implement a loop in which the 

companng images of solder joints, assuming the so der 30 primary image region ^ moved over a p^tennined num- 

joints are arranged vertically in the image, it may be that ^ rf ' rf ^ K ^ om , n ^ 

only the regions above and below the pnmary region should rf image region may & moved over or compared to all 

be included in the calculations. It is also not necessary for . ^ J . * . f Ko . ^ r . „„• „, 

. . , , . 4 . . image regions m the target image. In other cases, the pnmary 

neighbor regions to have a common boundary to the primary . ° °. L j i t * j u * 

utigLuui i 66 iui« uua . . * »j . image region may be moved only over a selected number of 

region. For example, one definition of neighbors could be 35 . ° ~. . / , ■ - X 

. , . r . . unage regions in the target image, 

regions that are 1 region away from the pnmary unage * 7 • ■ „ . , . 

re gj on Once decision is made in decision step 54 not to move to 

- ' . A , „ AO , , me next target image pnmary image, then processing flows 
Processmg then flows to step 48 where a target i mage eo ? ■.• u A. u * » u ■ c j u ^ *k 

• T i ■ * 4 j t * — r — ^ » t L to step 58 in which the best match is found between the 
region or bl ock is selected. In a manner similar to the . r . , 4 . . , 

f~*p tX 1 ~ u pnmary unage region and a target image pnmary region by 

selection of the pnmary unage region, regions which are40^ . 3 • * m ' ^ ' 

- ll . *u . * • ■ T^ur-rri=^n„ companng the scores computed in step 52. 
neighbors to the target image regio n can also be se lectedTtn 

this cSeln^ielghborfe^^^ respect to the target Processmg then flows to step 62 where decision is made 

image region. For example, a radius R ^may be us ed to as 10 whether more P 111113 ^ ima g e rc ^ ons should be P 1 ] 0 " 

determine^the^ghbor i^oi^yiucluarc-within^ pre- ccssecL J f more P rima yy ima S e regions remain to be 

defined distancTfron^ 45 P rooessc f *eo processing flows to step 64 where next 

the neighbor^loTlhT^rry image region and the target P nmarv ™8« r L c g l0 ° ^ selected. It should be noted that m 
image regions should be computed in the same fashi on. If cascs - cach subregion ofthe pnmary image is selected 

this is not the case, for insla^ce if the radius RHs^rTater or 45 a P rimarv subregion for processing. In other cases, 

equal to 2 (i.e. Rl>«2) and radius R2 is set equal 1 (i.e. however, it may be desirable or may not be necessary to 

R2-1), then only the neighbor regions one step away fiom 50 utilize each ima S e re S ion " tbe P™ 3 ^ ' m *&' B ? P rocess_ 

the primary image region and the target image regions ™S fewer ^ ^ subregions in the primary image region, 

should be considered in the computation. With R2 set to il ma ? be possible to increase the speed with which a 

zero, a comparison between one or more properties of the P rimar y and a ^ ""BP can be matched. After step 

primary image region and the target image region is made. 64 > processing retums to step 48 and steps 48-62 are 

^ Processing then flows to step 50, where one or more 55 rc P eatcd untl1 mcrc arc no morc ? nm ^ una S cs t0 P^ 55 ' 
properties of the seized pnmary image region and any h decision block 62, when decision is made that no more 

neighboring regions are compared to corresponding proper- P rimar y ima S e T& &°™ 10 P rocess exist > processing flows to 

ties in the target image region and any neighboring regions. ste P 66 10 which the best associated with each 

In step 52, a score indicating the difference between the pnmary image region are combined in order to provide a 

primary image region and its neighbors and the target image 60 ^re of how much deformation was required to map the 

regions and its neighbors is mmpiitP/i a nH pssnHateH with pnmary image to the target image. One way to calculate the 

the tarpetimage region^ scorejor instan^^ujdbe based amount of deformaUon * to add together the best score 

iirv ^rf^haracten^ticl nf luminance and positio n. If the associated with each primary image regions, 
radii Rl and R2 are 6, the score can be computed as a linear Processing then flows to step 68 in which an output is 

combination of the absolute difference in luminance and the 65 provided to a user. 

absolute difference in position of the primary and primary Referring now to FIG. 3C, a technique for computing a 

target image region. If radii Rl and R2 are greater than zero, score by computing a deformation distance between a query 
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image Q and a candidate or target image C is shown. used. Those ordinary skill will also appreciate that the 

Reference numerals q-Cg represents an image region of a number of rows need not be equal to the number of columns, 

target image and each of the reference numerals qi-q 8 Rather, the number of rows may be greater than or less than 

represents an image region of a query image. Thus q x the number of columns. 

denotes the i* image region of the query image Q while c y 5 Each block formed by the intersection of a row and 

denotes the j' image region of the target image C. column ^ ^ denoted ^ ?6XX WQere ^ l0 

In this particular example, three different axes of a Car- an index identifier for a particular block. For example, 

tesian coordinate system represent three different attributes, reference numeral 76aa denotes the block in the upper 

attribute 1, attribute 2 and attribute 3 of an image. Image left . hand aami of image 70> while reference numeral 76hh 

regions c^c 8 ^ ql -*q 8 are plotted along the axis. A distance 10 dcnotes ^ bbck m mc lowcf ^ hand comer of primafy 

between the attributes in the query image and attributes m a ^ JQ Thc block structure ^ n ^ refcrrcd to as a ^ 

candidate image are computed. The sum of the distances _ . , , c , , „ . , . 

, . " . r , ^ # . The particular number of rows and columns to be used m 

between the points, c^-c,* and 0,-0- are totaled and the *\ . . * ... . 

, . , . j tU i-Jr j- .~ any particular application to create image blocks may be 

minimum value is selected as the distance corresponding to / r , . r f . , . _ . - ° t , ' t 

. , t « t*u. *u u . - -.J™ selected in accordance with a variety of factors including but 

the best match. Thus the best match is denned as the image * * * 

, At _ . t r , r ^ » not limited to the desire to have distinctive or important 

having the least amount of deform ation and with respect to _ . , .. 4 , , 5. . 

, & . rTT 1 : ■ — 4 , t f features in the image delineated by one or more blocks. 

the query image. The distance is computed as a function ot _ . , c b , ' . . . _ 

!l r *u i\u ~ ** <■ #u Consider a field scene in which the important regions are 

the properties of the query image and the properties of the . t 4 , . . - V, T . - ^ . . , 

5 j . • ™ t j if * .i those corresponding to the sky and the field. It is undesirable 

candidate image. The function used can be, for example, the A . -j t/ ^ * * 

i_ i * j-« u. *u ^ Af ,. a - M . to have a very coarse end where many important features of 

absolute difference between the property of the is query ~ n . . J ' => . _ J 4 r , . 

, . A , , ..x j lt _ * r *u the image are merged together. The most extreme example 

image (denoted as prop(qi)) and the property of the j . , . 6 & . t & . . . . r 

f.j . • /j . 1 / \\ L n- i- ^ u \ of this coarseness is to consider the whole image as one 

candidate image (denoted as prop(q)) multiplied by a ^ . . f , . 

. , . s v r J block (1 row, 1 column). In this case, it would not be 
weight value 

„ r . " possible to distinguish the sky from the field. It is also 

Referring now to FIGS. 3D-3F , an mage 20 after the to have a ^ d where ^portant details in the 

divide and clasp steps are performed is shown as an image 2S arc separated by blocks. Consider the same fieldscene 

21. The prmiary tmage region and ne.ghbor regions are ^ ^ ^ each blade of grass. In thc 

shown as ^image region 23 Thus ^image region ,23 ,» provided most ^ Ae bMss &K ^ Qf individual 

from primary image block 23a and neighbor blocks . . 

23b-23p. Each of the lines 69 signify relative relationships ' « . . ■ • 

between regions. It should be noted that here relationships 30 Alu!oughprunay image 70 p here shown havmg a square 

between tte primary image block 23* and the neighbors sh »P e ' ' hose of ord } n 7 "V** "T" *™ « 

2323b-23«4 he, 23/, 23/Ind 23m are shown but it is also other shapes may also be used Althou gh primary image 7 0 

possibletodennerelaUonsbipsbetweentwoneighborblocks has beejui mded mto a plurality of sq^r jaon^ the 

fe e blocks 23c and 23*/) s ame size, those of ord inary skill in the art will appreciate 

t ir< j up • • that in some applications it may be desirable to use other 

Id F1GS.3E and 3F ' * pnmary image region 23 includes 35 ^ of sha ^ re ions ^ a differen , region reference 

primary image region diock is* and neignoon mad , zjc- in method ^ be ^ d _ For g 

in some applications 

an .mage are shown being deformed m to find a match ^ d ^ be ■ ^ or , he o{ 

between the primary image and a target image. It should be .• , . ,1 u ■ rc j u 1 Jl- _ 

..... .. F . 7 f .u • • • m^c ic j particular computations could be simplified by selecting 

noted that the shape of the primary region in FIGS. 3E and r, ... r . , • 1 . • 1 u 

. . * .u ■ ■ u blocks having a rectangular, a circular or a triangular shape. 

3F is not the same as the primary image region shown in 40 „ . ~ . ... . . ...... . 

FIG. 3D. In FIGS. 3E and 3F a primary image region 23' is • Target image 80 has likewise been divided into a prede- 

shown being moved between two different locations on a terrninedjiimbll ^flo^ and a pg dg jerm i ned n u mber of 

target image 71. The deformation of the image region 23' is columnsjt isja^t^onimontod^^ the 

indicated by a spring 25. The spring 25 starts where thc " me manner 35 ,he Penary image. In this case target image 

primary region exists in the primary image. To find a good 45 80 has Jj 6 numb6r of mws 306 mlumDS 35 P 1 ™^ 

match, the image region is moved from that spot over the unage 7U. 

target image 71. The spring 25 provides a visual indication vM>Tte primary image 70 is aligned with a target image 80. 

of the amount by which the primary image is deformed to 'This step can be performed before or after dividing the 

match the target image. The more the spring 25 is stretched images into regions or blocks. 

to find a good match in the target image, the more the 50 * D primary image 70, a first primary image block, 16bg is 

deformation. The spring may be implemented in a variety of selected. Next, a plurality of neighbors, proximate primary 

ways. For example the spring may be provided a function image block 76bg, are selected. In this case, the neighboring 

which itself includes a function of distance moved and other blocks are 76a/ 76ag, 76bf, 76cfc and 76cg. The primary 

properties. The spring technique may be implemented by image block 76bg and the neighboring blocks 76afc 76a& 

making a distance measurement and assigning to it a weight 55 7 &bf> 76c f> m( * ?6cg form a selected image region 

value. The weight is selected in accordance with a variety of After choosing a first primary region, a target image 

factors including but not limited to the emphasis which block, is selected. Also, a neighbor region is defied around 

should be assigned one parameter (e.g. a distance parameter) the target image block. In this particular example, a first 

vs. another different parameter (e.g. a color parameter). target primary block, &6gb, is identified and neighboring 

Thus, if it is desired to give greater consideration to spatial so blocks &6fb, 86g^ 86ha, and H6hb are also selected, 

distance than some other parameter, then the weight value is Thus, a target image region 88 is defined within the target 

selected accordingly. image. 

Referring now to FIGS. 4-4B, a primary image (PI) 70 It should be noted that in this particular example, the size 

has been divided into a plurality of rows, 70a-70h, and and shape of the primary image region 78 and the target 

columns, 71a-71k Although here eight rows and columns 65 region 88 have been selected to be the same. It should be 

are shown, those of ordinary skill in the art will appreciate noted, however, that it is not necessary for the size of the 

that fewer or greater than eight rows and columns may be regions 78 and 88 to be the same. For example, in some 
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embodiments it may be desirable not to identify any blocks If image 80 is considered the primary image and image 70 

neighboring block S6gb. In this case, only the properties of is considered the target image then the primary image 80 can 

region 76bg would be compared with the properties of be compared with target image 70 using the comparison 

region S6gb to determine a match. method described in FIGS. 3-3B. Based on this comparison, 

As shown, however, one or more selected properties of 5 a second similarity score can be computed. It is possible that 

each of the blocks within region 78 are compared with like the second similarity score (computed by comparing pri- 

properties of the blocks within region 88. mary i mage 80 with target image 70) will be different than 

Tlie region 78 is then moved to a next portion of the target the first similarity score (computed by comparing primary 

image. It should be noted that for an ideal match for primary image 70 with target image 80). 

image region 16b, the ideal location should be region S6bg 10 q ^ cxamplc can ^ ^ t0 mustra te this point. Let the 
m the target image. . V rcg j ons m j^ge 70 be all black. Let one region in the center 

It should also be noted that in some embodiments it may of { g0 ^ Mack and the rest white Ut lummance be 

be desirable or necessary to place constraints on the posi- ^ ^ tQ ^ ^ 70 ^ 

tions within target image 80 at which region 78 ^may be J/^ ^ 80> ^ the regions m ima^eTOwiU find 

placed. For example, as shown in phantom in FIG. 4B, it 1C r . . ^» * is-^.i JL — 'jji * — - 

may be desirable to not allow region 78 to be located such 15 inexact luminance match with the middle region in image 

that one of the neighbors falls outside of the boundary of the 80g^hejnatcb, between image Tff ana image W mjSE 

target image, as indicated by 89 in FIG. 4B. easels^ the, best possible match. Now, reverse the roles of 

The neighbor properties are matched when the selected ^ When image 70 is compared to image 80, 

region is compared with the region of the target image but on ^ L the black region wfll have a good lummance 

the weighting for the neighbor compares may be given less 20 match to regions in image 70. The rest will have a high 

weight than the primary image region. For example, in one luminance difference to all the regions in 70. The match 

embodiment, a Gaussian weighting scheme can be used. from 80 to 70 gives a low degree of; amilarity (or a high 

Thus, the properties of primary image region 76bg are match score). Thus, the match computation as irstands is not 

compared with the properties of target image region block symmetric. 

H6gb while the properties of blocks 76ag are compared with 25 It is sometimes desirable to have a symmetric measure- 
properties of block 86/b. The properties of block 76a/ are ment. For instance, his will help to insure that the same 
compared with the properties of block 86fo, the properties of images are returned as the result of a database search with 
76bf are compared witti the properties of block S6ga. The similar query images. For example, in FIG. 1A image 20 is 
properties of neighbor block 76c/ are compared with the used as the query image and images 22a-22/i are the images 
properties of block S6ha and the properties of neighbor 30 found ^ a resull of lne s^ch. if a second search were 
block 76cg are compared with the properties of blocks $6hb. performed using one of the images 22a-22h as the query 
The appropriate plug-in module (e.g., module 30 in FIG. (image 226 for example), a user would expect many 
1) determines what specific comparisons and properties are of ^ ^ mt ^ 22a and 22c-22/i to again appear in the 
performed. It should be noted that this will vary from results of the ^ st shown in nG 5 illustratc a 
application to application^ For example, m an image pro- 35 tcchniquc for making a symmetric measurement and, 
cessing application related to inspection of printed 1 circuit £ mcct ^ tations . 
boards, one set of comparisons and properties may be used ... nn t „ 
while in an application such as face detection, a second, Processing begins in step 90 where a first image (image A) 
possibly totally different set of properties and comparisons K matched to a second image (e.g., image B) and an 
may be used. The plug-in module may specify to commit ^ aggregate score is computed. The processing performed to 
properties of a first target neighbor, 86fo with the properties matcn A t0 ima g c B is the processing described above 
of a first primary image region neighbor, 76ag. The plug-in m conjunction with FIGS. 3-4B. 

module may also specify to compare the relative properties Processing then flows to step 92 in which the second 

of blocks with the primary and target region. For instance, image (i.e. image B) is matched to the first image (i.e. image 

the plug-in-module may specify to compute a relative prop- 45 A) and a second aggregate score is computed using the 

erty (such as relative luminance) between region 76bg and processing steps described above in conjunction with FIGS. 

766/ This relative property can be compared to the relative 3-4D. 

property between the corresponding target blocks S6gb and Next, as shown in processing step 94, the scores from the 

H6ga. two searches are combined to provide a composite score. 

If the properties are not consistent, then this factor would 50 The scores may be combined, for example, by simply 

be taken into account when computing a total score for the computing a mathematical average or alternatively, the 

match at that particular location. scores may be combined using a different mathematical 

The aggregate score given a primary image region, a technique which includes weighting function or other tech- 
target image region, and a set of properties is computed as nique to emphasize one of the scores, 
a function of the difference between the each of the actual 55 One could also retain the scores for each property in steps 
values of the properties in the primary and target region. A 90 and 92 of FIG. 5. Step 94 could average the score for each 
simple example of the function is function which increases property independently and then combine the result into an 
linearly by the difference in the primary and target proper- aggregate match for image A to B and B to A ^. 
ties. Referring now to FIGS. 6-6 C, an image 100 includes 

Referring now to FIG. 5, a the processing steps to makeHjO images regions 100a-100/?. Semantically, the image rep re - 



a symmetric measurement between two images is shown. 
The method described in FIGS. 3-3B computes how similar 
a primary imag ejs to a target image. In F1U. 4, a comparison 
was made between the primary image 70 and the target 
image 80 (i.e. primary image 70 was compared with target 
image 80). Based on this comp arison, a similarity score was 
computed. 



seats a picture of a square having portions or lines 102a 
-102g as shown. Each of the image portions 102a-102g 
which fall within a single image region are combined or 
collapsed to thus represent all of the information included in 
65 each image portion in a compact manner within the image 
region. For example, image region 1006 contains image 
portions 102a, 1026. Image portions 102a, 1026 are thus 
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combined and represented as an image portion 104a (FIG. 
6A) within image region 100b (FIG. 6A). 

Likewise image portions 102c, 102a* are combined and 
represented as image portion 104b in image region 100/t; 
image portions 102^ 102/ are combined and represented as 
image portion 104c in image region 100;; and image por- 
tions 102& 1Q2/i are combined and represented as image 
portion 104d in image region lOOe. Thus image 102 may be 
compactly and efficiently represented as image 104 com- 
prising image portions 104a-104<£ Image 104 is used to 
perform searching within an image database such as data- 
base 18 (FIG. 1). The above process can be performed by an 
appropriate one of optimizers 30. 

Each image within the database 18 is likewise compactly 
A Represented such that when two images are compared, a 
relatively large amount of image information is contained 
within a relatively compact representation thus allowing 
rapid comparisons to be made. Also, by utilizing the "col- 
lapsed image" approach, an image can be stored in a 
relatively small amount of storage space. 

Referring now to FIGS. 6B, 6C a collapsed query image 
comprising image portions 106a— 106a* can be compared 
with an exemplary collapsed database image 108 comprising 
image portions 108a-108/. The closeness of the match of the 
query image 106 to the database image 108 determines tjie 
value or score assigned to the database image 108 which 
represent the closeness of the match between the query 
image and the database image. To compute a score, an 
assessment is made to determine how much deformation of 
the query image must take place to match the query image 
106 with the database image 108. 

For example, query image portion 106a appears in the 
same image segment as database image portion 108a and 
thus no deformation of the query image is required to match 
the two image portions. Query image portion 106fc must be 
deformed or moved in a downward direction by one image 
segment to thus align with database image segment 108c. 
The same matching steps are carried out for each portion of 
the query image portions 106a-106d with respect to each 
segment and portion of the database image 108. Once each 
of the query image portions 106o— 106a* have been tested for 
a match then the process is repeated for the next target image 
which may for example be retrieved from an image data- 
base. 

^ FIGS. 7-7B illustrate the image processing and matching 
techniques described above in conjunction with FIGS. 1-6C 
in the context of a trademark searching system. It should be 
appreciated, of course, that the concepts described in con- 
junction with FIGS. 7-7B can also be used in a variety of 
other applications including but not limited to inspection 
systems, face recognition systems, scene classification and 
image search systems, biotech analysis and search systems, 
and fabric/textile management and search systems. 

5V Referring now to FIGS. 7-7B, in which like elements are 
^ provided having like reference designations, an output dis- 
play screen 130 includes a tool bar 132, a query image 
choice region 134, a selected query image region 136 and a 
results image region 138. Below each image 138a-138/t 
displayed in the results region 138 is a corresponding value 
139a-139/i cor responding to a measure or score indicatin g 
th e similarity (ox jnalfih) ^ f ^ f 'h e images 138a-138 /i 
to the query ima ge_H6. As can be seen in FIG -^Lomage. 
138a identically matc hes querx lsiage^6Und.thiis .receives 
a sc ore of zero (0) mdicatmgj.best.possiblejna[ch between 
the quervlmafie 136 and a target ima gg^ wEch j nav, for 
example ^ be stoied in a Storage device such as an image 
database system. 
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It should be noted that the selected query section 136 is 
not limited to just one image. Hie results section 138 can 
contain 1 or more results, not necessarily eight images as 
here shown. Images in the results section 138 can also be 
selected and put into the selected query region 136. 

In this particular example, a display of three images 
134a-134c as query choices is shown. The rightmost image 
134c (the triangle with the swirl) was chosen as the query 
image and placed in the query location 136. A search system 
using the techniques described above in conjunction with 
FIGS. 1-6C matched the query image 136 to images in an 
image database. A status bar 140 at the bottom of the display 
screen provides a real time or pseudo-real time qualitative 
indication of the number of shows how many images the 
system has examined in the database. In this example, the 
status bar 140 is approximately l A the distance from the start 
point, meaning that the system has matched the query image 
to 33% of the database. 

The system orders the whole database in terms of how 
similar each database image is to the query image. It 
displays the results in order of their computed visual simi- 
larity to the query image. In FIG. 7, the eight result images 
with the lowest scores are shown in the results section 138. 
In the case of FIG. 7, the syste m computed the sim ilarity 
25 scoresasjlescribed in conjunction with FIGS. 1-6C between 

the q^ipg^magp jflfi anH all thp. imagp.Q in thr. fjataha^ The 

scores were associated with their corresponding image in the 
database. The images in the database can thus be sorted or 
ordered based on these similarity scores. These scores values 
30 are noted as 139 in FIG. 7. It is not necessary, however, to 
use the method described in conjunction with FIGS. 1-6C to 
compute similarity. Other algorithms may be used to com- 
pute the similarity between the query image and the images 
in the database. The images in the database, thus, can be 
ordered based on these different types of calculations. 

As can be seen from FIG. 7, the search results of those 
images determined by the search system to most closely 
match the query image are displayed. In this particular 
example, the closest eight (8) matching images from the 
database are displayed to a user along with the similarity 
scores. An information button 141 next to each result image 
138 provides a means to retrieve information related to the 
image. For instance, in this case the images correspond to 
trademark images. Thus, the related image information may 
include the name of the entity who owns and or who filed the 
mark, the filing date of the mark, the date of registration of 
the mark (if any), and the class of goods with which the 
marked is used. 
In the case of an inspection system, such as a printed 
50 circuit board inspection system, the images may correspond 
to circuit components or printed circuit boards. In this case, 
the related information could correspond to information 
about the component type or include manufacturing infor- 
mation related to the printed circuit board or circuit com- 
55 ponent. 

It should be noted that the display may be dynamically 
updated during a search. By comparing the position of 
images 138a-138/i in FIG. 7 to their positions in FIG. 7 A, 
a so-called "ripple feature" of the search system is revealed. 
As mentioned above, FIG. 7 showed the state of the image 
search after processing 33% of the images in the database. 
As can be seen from the status bar 140, FIG. 7 A shows the 
state of the image searching system after processing about 
66% of the images in the database. By dynamically display- 
ing the images, the top eight images are periodically updated 
(e.g. after every few seconds) to show the best images at any 
one discrete instant in time. 
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For example, by comparing the order of the result images search (e.g., prior to completion of all images in a database) 

from 138 in FIG. 7 to the order of the result images 138 in since any of the resulting images 182a-182h can be selected 

FIG. 7 A, it can be seen that image 1386 has moved from its and used as starting points for a new search. For example, as 

position in FIG. 7 as the second best image (after about 33% shown in FIG. 8, image 182c is selected and an image 

of the images to be processed in the database have been 5 processing system (e.g., system 10 in FIG. 1) can use the 

processed) to a position in FIG. 7A as the fifth best image image 182c as the query image to perform the search, 

(after about 66% of the images in the database have been j t mav a iso be desirable to stop the search so that any 

processed). This indicates that three images which have been resvll images 182a-182/i that are interesting to the user can 

processed in the middle third of the database match the be stored. In FIG. 7, element 132/ corresponds to a pause 

query image more closely than image 1386 which was found 10 button that may be used to stop the processing. Engaging 

in the first third of the database. ( e> g. "clicking" or pressing or selecting the icon with a 

Such image updates can be provided to a user in real or pointer) the icon 132/ resumes the search, 

pseudo real time. A user viewing this dynamic update Furthermore, by allowing the search to be interrupted, a 

process in which some images move from one location to user watching the visual display can stop tbe search if a 

another, some images move off the display, and other new 15 malcn ^ f ounc j without waiting for the image processing 

images appear on the display the effect appears as a "ripple system to complete a search of the entire database thereby 

effect." decreasing the amount of time needed to satisfy a query; in 

Referring now to FIG. 7B, a flow diagram explaining the this case finding a similar or infringing trademark, 

steps required to implement the above described "ripple j t should be appreciated that although the images 

effect" are shown. It should be noted that this particular 20 ig2a-lS2h in FIG. 8 are here shown as arbitrary shapes, the 

example, (as well as the description provided above in samc principles apply, for example, to processing and 

conjunction with FIGS. 7 and 7A), assumes that a set of inspection of printer circuit boards and integrated circuits, 

images are stored in an image database. However, the p or CX ample, in a printed circuit board processing system, 

concepts described herein above and below can be used me i ma g e s stored in the database could correspond to 

regardless of whether the images are stored in a database or 25 images of satisfactory and/or unsatisfactory solder joints or 

other storage device. It should also be noted that the example component placements. In this case, an image of a printed 

in FIGS. 7 and 7Ahave only one query image. The concepts circuit board (or portion thereof) being fabricated or 

described can be used with multiple query images as inspected would be compared to the previously stored 

described hereinbelow in conjunction with FIGS. 11-11B. images in the database. By matching the image of interest to 

Processing begins in block 160 where a query image and an image in the database, a decision as to whether a 

a set of target images are selected for processing. Typically, satisfactory/unsatisfactory solder joint or component place - 

a predetermined number of target images (N) are selected for ment exists can be made. 

processing. Processing then proceeds to block 162 in which Referring now to FIGS. 9-9F, the manner in which a 

the query image is compared with the subset of the target ^ image may be obtained is shown. The query image 

images. The comparison may be accomplished using the may be an existing image in the database, as shown in FIGS, 

techniques described above in conjunction with FIGS. 7 aac j 7 a. (FIG. 9 shows the query image used to perform 

3-3D. T he size of the subset of time target i mages mav be mc search.) The query image may be scanned in from a hard 

denoted aX^-x" where x is less than a predetermined number cop y or captured from a camera. It may be hand drawn as 

of tar get images N. (Note, that it is not necessary to have "x" ^ sn own in FIG. 9A. The query image may be a manipulated 

be the same value across all iterations of the loop 162-172). version of the known image. For instance, FIG. 9B shows 

A similarity score is associated with each target image in the piG. 9 rotated in the image plane clockwise 90 degrees. FIG. 

subset. 9C shows FIG. 9 flipped about a vertical axis. FIG. 9D 

Processing then proceeds to block 164 in which the shows FIG. 9 contrast inverted. The query image may also 

processed target images and their respective scores are 4$ be one of the resulting images after an initial search is 

stored in a list of processed images. Next as shown in block performed (as discussed above in conjunction with FIG. 8). 

166, the list of processed images are sorted by their simi- Referring now to FIGS. 9 and 9E-F, the manner is which 

larity score. one or more portions of a query image may be emphasized 



r U In block 168, the set of processed target images are moved 
^Mrom the database of target images and the value of_N is^ 
decreased. In this exam ple^ thej^maaber of images left t o 
proc ess denoted as N, is decreasej ih y_a vahie^corresnondi ng 
to_x^ Processing then proceed s- to block 170 where the be st 
M images are displayedto the user. P rocessing then flows to 
decision block 172 where decision is made as to whether 
there are any more target images to be processed. If there are 
more target images to be processed, processing flows to 
block 162 and the processing in block 162-170 is repeated 
until there are no more target images to be processed. — 1 



is shown. FIG. 9 shows a query image 190. Portions of this 
50 image may be emphasized according to their importance. 
FIG. 9E shows two regions 192, 194 marked as significant. 
The region 192 corresponds to the swirl area of the image 
190. This indicates that the swirl pattern of the image should 
be more heavily weighted than the rest of the image for the 
matching process. The region 194 corresponds to the trian- 
gular area of the image 190. This indicates that the triangle 
shape at the scale of region 194 of the image 190 should be 
considered to be more heavily weighted than the rest of the 
image for the matching process. 



Referring now to FIG. 8, a visual display 180 of images 60 Also a relative relationship can be defined between each 

\S2a-lS2h resultant from an image search is shown. It of the selected image regions; such as between regions 192, 

should be noted that an image search may be interrupted or 194. An example of a relative relationship between regions 

stopped before completion in which case images cone- is a spatial relationship. In the case of FIG. 9E a region 

sponding to result images at that point in time will be similar to 192 should be "below" a region similar to 194. If 

displayed. 65 two regions in a target image are found to match well with 

It may be desirable to be able to stop the search and view regions 192 and 194 and the previously defined relative 

the result images at times prior to the completion of the relationship is satisfied in this target image, then this match 
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should be more heavily weighted than one where the relative 
relationship is violated. 

It should also be noted that portions of the image may be 
removed or erased to completely de-emphasize a feature. 
For instance image 196 in FIG. 9F corresponds to image 190 
with the triangle portion completely erased. Thus image 190 
can be modified to provide image 196 which shows only the 
"swirl" section. The image 196 can then be used as the query 
image. 

Referring now to FIGS. 10 and 10A, the processing which 
takes place to process an image using a selected portion of 
an entire image is shown. Processing begins in step 200, 
where a subregion of a primary image ("primary subregion") 
is selected. As shown in FIG. 10A, an entire image 220 has 
a subregion 222 selected. 

Processing then flows to decision block 202 where deci- 
sion is made as to whether the target or the primary 
subregion should be resized. If decision is made to resize 
one of the images (e.g. the primary subregion), then pro- 
cessing flows to block 204, where the one or more resized 
images are generated. It should be appreciated that in some 
applications it may be desirable to resize the target image 
and the primary subregions such that they are the same size, 
in which case processing then flows to block 206 and steps 
performed in conjunction with FIGS. 3-3D are performed. 
In other applications, however, it may be desirable to 
generate a so-called image pyramid. 

Processing then flows to decision block 207 in which 
decision is made as to whether any more resized images 
remain to be processed. If more images remain to be 
processed then processing returns to block 205. If no more 
images remain to be processed then processing ends. 

In the image pyramid case, each image in the image 
pyramid corresponds to the primary subregion at a different 
scale. In this case, each scaled primary subregion is com- 
pared to the target image and the origin of each of the images 
in the image pyramid are aligned at a particular location with 
the target image. Then processing to compare the images in 
accordance with steps explained above in conjunction with 
FIGS. 3-3D is performed as shown in block 206. It should 
be understood that the origin of each of the images in the 
image pyramid can be placed at multiple different locations 
in the target image and a comparison between the pyramid 
image and the target image is made in accordance with the 
processing explained above in conjunction with FIGS. 
3-3D. This is done by proceeding through steps 208-212 in 
FIG. 10 for each pyramid image. The processing performed 
in blocks 208-212 is the same as that performed in blocks 
214-218 and will be explained below in conjunction with 
blocks 214-218. 

If in decision block 202 decision is made to not resize an 
image, then processing flows to block 214 where the origin 
of the selected region is aligned with a reference point in the 
target image are selected. 

Processing then flows to block 215 where certain condi- 
tions and constraints are specified. For example, boundary 
conditions may be specified. The boundary conditions can, 
for example, place limits on the positions at which region 
222 may be placed relative to the target image. For example, 
one boundary condition may be that no portion of region 222 
may lie outside any region of the target image. 

When selecting the origin, care should be taken to not 
select a reference point which causes the subregion (e.g. 
region 222 in FIG. 10A) to move outside the boundaries of 
the target image. Such a constraint may be useful to decrease 
the amount of time required to match the selected subregion 
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to the target image because many of the possible reference 
points in the target image will be invalid. Usually, these 
invalid reference points are in the top, top-right, and right 
portions of the target image, forming a rotated L shape. 

5 In some embodiments it may be desirable to specify the 
boundary conditions prior to selecting a reference point. 
Regardless of the order in which steps 214 and 215 are 
performed, the boundary conditions are compared with the 
selected reference points to determine whether a selected 

1Q reference point is valid. 

In this particular example, subregion 222 is assumed to 
have an origin at its upper left-most corner. It should be 
appreciated, however, that other origins could also be 
selected for subregion 222. For example, the lowermost 
right hand comer of subregion 222 could also serve as the 

15 origin. Thus, although in this example, the origin is assumed 
to be in the upper left hand corner, it is not necessary for this 
to be so. 

Once the origin of the subregion 222 and its reference 
point in target image 224 are identified, it is then possible to 

20 place subregion 222 at that reference point in in target image 
224. In this particular example, the subregion 222 is shown 
having its origin located at the same position as the origin of 
the target image 224. 

If the subregion and the target image are not the same size, 

25 it should be noted that it is not necessary to place the origin 
of subregion 222 at the origin of the target image 224. The 
origin of subregion 222 may be placed at any location within 
target image 224 as long as all other constraints (including 

3Q boundary conditions) are satisfied. 

It should be noted that in some applications it may be 
desirable to select more than one region in the primary 
image for matching to the target image. This is illustrated in 
FIG. 9F where regions 226, 228 are selected. In the case 

35 where two image regions are selected, in steps 214 and 215, 
reference points in the target image for each of the sub 
regions must be identified and constraints between proper- 
ties of the two selected image regions should be taken into 
account. 

40 Processing then flow to block 216 where the processing to 
compare the images in accordance with steps explained 
above in conjunction with FIGS. 3-^3D is performed. The 
data from processing with selected reference points is stored 
in a storage device (e.g. a processor memory) and processing 

45 move to decision block 218 where it is determined whether 
another reference point should be selected. 

If in decision block 218 decision is made that more 
reference points remain to be processed, then processing 
returns to block 215. If no more reference points remain to 

50 be processed then processing ends. 

The query to image processing system 12 (FIG. 1) may 
contain more than one image. FIG. U describes a general 
technique to match multiple query images to a set of target 
images. FIGS. 11A and 11B describe two methods to com- 

55 pute an aggregate score for each target image and to provide 
output to a user. 

Turning now to FIG. 11, processing begins in step 230 
where a first primary image is selected. Processing then 
flows to step 232 where a first target image is selected from 

60 a storage device. In Step 234, a similarity score between the 
primary image and the target image is computed and record 
consisting of a primary image identifier, a target image 
identifier and the score associated with the target and pri- 
mary image is stored in a storage device which may be a 

65 temporary storage device (e.g. a random access memory — 
RAM) or a permanent storage device such as a magnetic 
tape or compact disc read only memory (CD ROM). 
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Processing then flows to decision block 236 where deci- 
sion is made as to whether more target images should be 
processed with the same primary image. If decision is made 
to process more target images, then processing flows to 
block 240 where a next target image is selected and a 
similarity score between the primary image and the next 
selected target image is again computed. The primary and 
target image identifiers and the associated score are again 
stored in a storage device. This loop is repeated until it is not 
desired to select anymore target images. 

Processing then flows to decision block 238 where deci- 
sion is made as to whether more primary images should be 
processed. If a decision is made to process more primary 
images, then processing flows to Step 242 where a next 
primary image is processed and blocks 232 through 242 are 
repeated until it is not desired to process anymore primary 
images. Processing then flows to block 244 where a score is 
computed and an output is provided to a user. The particular 
manner in which these scores could be computed is 
described below in conjunction with FIGS. 11A and UB. 

FIG. 11A describes a method for calculating an average 
score for each target image over all the primary or query 
images. This is one of many techniques that may be used in 
step 244 of FIG. U. In FIG. UA processing begins in 
processing block 246 where a first target image is selected. 
In block 248 all records with the target id are retrieved from 
the storage device. The scores associated with that image are 
added and divided by the number of primary images. 

Processing then flows to block 250 where the aggregate 
score is associated with the target image. Processing then 
flows to decision block 252 in which decision is made as to 
whether more target images should be processed. If more 
target images should be processed, then processing flows to 
block 254 and blocks 248 and 250 are repeated. This loop is 
repeated until there are no more target images to process. 
When decision is made in decision block 252 that there are 
no more target images, then processing flows to block 256 
in which the target images are sorted according to their 
aggregate score. Processing then flows to block 258 in which 
the sorted images and associated scores are provided to the 
user. Such output may be provided in the form of a visual 
display, an audio display or a printed display. 
1 FIG. 11B describes a different approach to jxmabinin g 
scor es from multiple query jmages . In FIG. UB,~processing 
begins in block 260 where lfirsTprimarx image is^ sglected. 
Processing then flows to block 262 whe re all records w ith 
the prim ary image id are retrieved. From this set of records 
all targ et images are sorted according to the. scpre associate d 
with the p rimary imag e, lnblock 264, the list ot sorted target 
images and their scores which are associated with the 
primary image are stored in a storage device. In decision 
block~266, decision is made as to wnetneF more primary 
images should be processed. If decision is made to process 
more primary images, then processing flows to block 268 in 
which a next primary image is selected and processing then 
flows to blocks 262 to 264. 

If in decision block 266 a decision is made to not process 
more primary images, then processing flows to block 270 in 
which a threshold number is selected. In block 272, target 
images that are in the first N positions in all the lists 
associated with the primary images are identified. In block 
274, the resulting subset target images are sorted by best 
score or best average score and provided to a user. The 
information may be provided in the form of a visual display, 
an audio display or a printed display or using any other 
display technique well known to those of ordinary skill in 
the art. 
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There are many other techniques to compute a similarity 
score based on a multiple set of query or primary images. 
For example one may match each primary image to each of 
the target images individually. A score is then associated 

5 with each target image and this record is put into a list. The 
list can be sorted based on the computed scores. The system 
should return a non-duplicative set of images that have the 
lowest individual score to at least one primary image. 
Rather than trying to combine multiple scores derived 

10 from multiple primary image, one may calculate the com- 
mon characteristics across the primary images to produce a 
new "condensed" query image that embodies those charac- 
teristics. The matching may be performed using this new 
query image and a target as described in FIGS. 3-3B. The 
output is a similarity score between the condensed query 
image and the target. Target images may be sorted based on 
their respective scores. 

Alternatively still one may match multiple query images 
to each other before processing the target images. The goal 

2Q is to find the system parameters that find the most consistent, 
best similarity scores calculated between query images. In 
other words, the goal is to find the system parameters that 
best explain the similarities between the query images. 
These system parameters and one or more of the query 

25 images may be used to calculate a similarity score to the 
target images. 

It should also be noted that the query may be refined using 
positive examples. The results may be calculated by one or 
more query images. The system parameters may be refined 
30 by having the user choose images from the result set as 
"positive examples". One way to refine the parameters is to 
alter the them such that the resulting measure of similarity 
gives the lowest (or best) possible scores to the positive 
examples. 

35 Alternatively, it should also be noted that the query may 
be refined using negative examples. The results may be 
calculated by one or more query images where now the user 
may specify both positive examples (images that fit a 
particular criteria or embody the perceptual concept) and 

40 negative examples (images that do not embody the percep- 
tual concept). One way incorporate positive and negative 
examples is to alter the system parameters such that the 
resulting measure of similarity produces a maximal differ- 
ence between the scores for the positive examples and the 

45 negative examples. Another method is to compute the com- 
monalities between the positive examples and then to 
remove any features contained in the negative examples. A 
new query images that embodies the characteristics of the 
positive examples and does not contain the characteristics of 

50 the negative examples may be reduced and used to perform 
the matching. 

f^jy Referring now to FIG. 12, the steps for first sorting images 
by text and then by visual similarity as shown. Processing 
begins in block 276 in which a primary imag e is retrieved. 

55 In block 278, thejext associ ated with the iSage is also 
ret rieved . This text may be retrie^eTfro^^oTe^'ififormation 
or by data input by a user. In block 280 all other images with 
at least one common associated word are retrieved. In block 
282, these images are grouped into a list of target images. 

60 This completes the retrieval by text step. The next step is to 
sort the list of target images by their visual similarity to the 
query or primary image. In block 284, a primary image is 
compared to each of the target images in the list The 
comparison is made using the techniques described above in 

65 conjunction with FIGS. 3-3D. 

In block 286, the similarity score with each target image 
in the list is stored. In block 288, the list of target images is 
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sorted by similarity score and in block 290, an output of the ensures that the target images that are returned are in some 

sorted list is provided to a user. way related to the query image. Unfortunately, the resulting 

Referring now to FIG. 12Aan exemplary data format 292 »t of images ma y be extremely long and unsorted. Without 

is shown. In this particular example, data string 292 includes v*™ 1 similarity matching, a user would have to look at each 

an image identification field 294, an image data field 296 and 5 image to find the best match. Vi sual sim Oanty-matching 

associated text fields 298. The image identifier field 294 may sorts Jheiesuliingje^^ 

be provided for example as an alphanumeric or a numeric to the que^yjmage. Thus, me^mostjjsuaL : ^ 

identifier which indicates the position of the image on the ima ges-are given priorityuBihejist over less similar im ages, 

storage device on which the image is stored. For example, This last ste PS brin gs *ne most salient images to the attention 

the image data may be stored on a magnetic tape, a magnetic i° of the user first. The user may usxjhejimil^^ 

disk, a read only memory, a CD ROM or any other storage d etermine when to stop looking at the list o f images. If the 

device well known to those of ordinary skill in the art. The scores get toohigh or there is a large jump in scores, this 

image ID may or may not be related to the order in which information may signal to the user that images after that 

the image is stored on the disk and in some embodiments the point may not have much visual similarity with the query 

image ID field may be omitted. Image data field 296 15 image. 

corresponds to the values (e.g., digital values) which when FIGS. 12-12D describe the use of text and visual simi- 

properly interpreted result in the display of an image. The larity matching as a two stage process to compare a query 

value is made to depict a color image or a gray skill image image to a set of target images. The two searches, however, 

or any other type of image representation known to those of may be integrated. Target images may be given a score 

ordinary skill in the art. 20 which is a combination of a text match and a visual 

Associated with each image data field 296 is a text field similarity score. 

298. The text field 298 includes a textural description of the Referring now to FIG. 13, a circuit component 530, 

image provided by the image data 296. Thus in this partial- having a lead 332 coupled to a circuit line 334 via a solder 

lar example, image 296a is described by text 298a as being joint 336 is shown. Such a circuit component is placed on a 

a triangle swirl or geometric object. The particular text 25 printed circuit board and coupled to the circuit line which is 

associated with each image is typically subjective and typically fabricated at part of the printed circuit board via an 

descriptive of the particular image. Image 2966 is an image etching or deposition technique. 

of an apple and thus the text 2986 associated with image Th e particular manner in which the circuit component 

2966 is apple, fruit and food. Other associated text may also 330, circuit line 334, and lead 332, as well as the soldering 

be associated with the apple such as the color red for technique used to couple the lead to the circuit line, may 

example. In addition, he text information may not consist of have an impact on the specific processing which takes place 

a meaningful word. Often codes are used to describe images, during an inspection process. However, during the inspec- 

such as the numeric coding scheme used by the Patent and tion of the circuit component, different features or charac- 

Trademark office to describe trademark images. An example ^ teristics may become more or less important depending upon 

of such a code is the text string 01.01.03 which is known to the manner and/or technique used to mount the circuit 

be associated with a five pointed star. component, couple the circuit component to the circuit line, 

In example in FIG. 12 A, the data format 292 illustrates the and fabricate the circuit line, 

fields 294, 296 and 298 in a contiguous manner. In some i n this particular example, inspection of the gull wing lead 

applications, however, the information may not be stored ^ 332 and solder joint 336 is described. In this particular 

contiguously but rather may be associated by links from one example, it is recognized that the relative brightness 

field to the next. Thus any storage device the image data may between predetennined regions at and proximate to the 

be stored in one portion of the memory and the text field solder joint 336, can be used to verify the joint as an 

associated with that image data may be stored in a different acceptable or an unacceptable solder joint. Specifically, a top 

portion of the memory. 45 surface of circuit component 330 reflects a certain amount of 

FIG. 12B shows the query image (an apple) while FIG. light. The angled portion 332a of lead 332 reflects less light 

12C shows the first eight images in the database which had and thus appears darker than region 330a. Similarly, flat lead 

been annotated with the word "apple". The results are shown portion 3326 appears brighter than lead portion 332a and a 

in order of how they appear in the database (which has no first portion of solder joint 336a appears darker than lead 

bearing on how visually similar they are to the query image). 50 portion 3326. Solder portion 3366 appears brighter than 

FIG. 12D shows the set of images with the annotation region 336a and solder region 336c appears darker than 

"apple" sorted by visual similarity to the query image (FIG. region 3366 due to the angle at which the solder contacts the 

12Q. The best eight images are shown in FIG. 12D. It circuit fine. Circuit line portion 334a corresponds to a flat 

should be notice in FIG. 12C that images 315c and 3154, portion of the circuit line 334 and thus reflects light at a 

which are very similar to the query image 310, are in the 55 different angle than solder region 336c and thus appears 

third and fourth positions. After sorting by visual similarity brighter than solder region 3366. Thus, when inspecting the 

they appear in the first and second positions in FIG. 12D. solder joint 336, the relative luminance of each of the 

The reordering of 315a and 3156 after visual similarity regions can be used in the inspection process and it is these 

matching should also be noticed. These are less visually relative luminance characteristics which can be specified in 

similar to the query image. Image 315a is moved to the 60 a plug-in module to thus expedite the image matching 

fourth position and image 3156 is moved to the third process. 

position as shown in 12D. Image 315g which is not at all Referring now to FIG. 14, the steps to process an image 

visually similar to the query image is displaced entirely from m a circuit component inspection system are shown. Pro- 

the top eight images. This image does not appear in FIG. cessing begins in step 340 where an image is captured. 

12D. 65 Referring briefly to FIGS. 13 and 13 A, the image can be a 

The use of text based search with visual similarity match- portion of an entire printed circuit board containing a 

ing has several good attributes. The text based search component and the leads from the component as illustrated 
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in FIG. 13A(note in FIG. 13Aonly one lead is shown). The look very visually different from the primary image. If the 

image would thus include a top view of circuit component primary image has a defect, it should look dirferent than the 

330, a top view of the lead 332, a top view of the solder joint other leads from the component. Leads from one component 

336, and a top view of the circuit line 334. Processing then may be compared also across different boards, 

proceeds to step 342 where the existence of the soldered 5 The target image does not have to contain both the lead 

component is verified. This can be accomplished by simply and the joint. Just the solder joint as shown in 339 of FIG. 

matching the circuit component region of the image portion 13A could be processed. Thus, the score value would 

of FIG. 13a to a stored image of a circuit component. Next, represent the closeness of the match between image 339 and 

as shown in step 344, the circuit connection points (i.e., the the target image which is the image of a good solder joint, 

circuit leads 332) are identified. The leads can be identified 1Q Thus, a perfect match would indicate that image region 339 

by matching a circuit component image to the captured is also an image of a good solder joint If the match between 

image and knowing a priori where the leads are with respect the target image and the primary image 339 is not good, this 

the circuit component. indicates that the target image is not an image of a good 

Processing then proceeds to step 346 where each lead and solder joint. Note also that views other than the top view of 

^ solder joint is seen as a primary image. The target image 15 a circuit component could be used as bom primary and target 

may be another stored image (or images) of a good lead and images. 

solder joint or a synthetic or composite image of the lead and Referring now to FIG. 15, in block 350 an unpopulated 

joint. The target lead/joint is processed in accordance with printed circuit board having electrical circuit lines etched, 

the steps explained above in conjunction with FIGS. 3-3B deposited or otherwise provided thereon is provided to a 

using the lead phig-in-module (as referred to in the discus- 20 solder paste station as shown in block 352. The solder paste 

sion of FIG. 13). The output value (i.e., the score) from the application station may be provided for example as a screen 

processing of step 346 is compared with a threshold value. printer or any other device well known to those of ordinary 

If the score from processing 346 is greater than a predeter- s ]qU j n the art to apply solder paste to a printed circuit board, 

mined threshold value, then the lead/solder joint is classified i n some embodiments, the solder paste may be applied by 

as a bad connection. (We are assuming that a low score 25 hand. Regardless of the particular manner or technique used 

corresponds to a good match, where a score of 0 is a perfect to apply solder paste to the printed circuit board, the solder 

match). Processing then ends. paste is applied to predetermined regions of the printed 

It should be noted that the threshold may have to be circuit board. The solder paste should be applied in a 
adjusted for different types of leads. Leads may vary in predetermined amount within a given range. Processing then 
many aspects such as shape, size, and pitch. The threshold 30 flows to block 54 in which a solder paste inspection system 
value may also change depending on the goals of the inspects the solder paste applied at the predetermined 
manufacturer. The manufacturer when setting up a produc- regions of the printed circuit board. Hie solder paste inspec- 
tion line may want to find all the bad leads at the expense of tion system captures one or more images of the regions of 
some false positives. In this case threshold in 348 should the printed circuit board at which solder paste has been 
biased towards the low side. The manufacturer can analyze 35 applied and at which inspection is necessary. The image 
where and why the bad leads occur and change the produc- captured by the paste inspection system may for example be 
tion process to compensate for these errors. (This will be an image of the entire printed circuit board in which case 
explained more fully in the text in conjunction with FIG. selected image regions of the printed circuit board image can 
14). Alternatively, when the production process is in full be identified and compared to target images. The target 
swing, the manufacturer may want to bias the threshold 40 images correspond to images of similar printed circuit board 
towards the high side to accept all good leads at the expense regions in which the solder paste has been properly applied, 
of missing a few bad leads. By comparing the captured image region to the target region 

It should be noted mat in step 346, the target images could and performing the imaging processing techniques 
be pictures (real or composite) of bad leads. It would be ideal described herein above in conjunction with FIGS. 3-14, a 
if this set of target images spanned the whole class of lead 45 determination can be made as to whether the solder paste 
and joint defects. It would also be desirable, if each of the applied in block 352 was properly applied in each of the 
target images had information regarding what defect it appropriate regions of the printed circuit board. If decision 
portrays. In this case, a new lead such as image portions is made that the solder paste was not properly applied in one 
332-334 in FIG. 13A are compared against the target or more of the examined regions, then the printed circuit 
images. If similarity is high (a low score) between the so board is returned to block 352 where the solder paste is 
primary image and any target image, then the primary image reapplied in each of the regions in which it had not been 
should be classified as defective. The system should record properly applied in the first instance. Thus, blocks 352 and 
not only the best similarity score, but also the target image 354 are repeated until the paste inspection system deter- 
that produced that score. This information will tell the mines that the solder paste has been properly applied in each 
system not only that the lead and joint are bad, but also why 55 appropriate region. 

they bad. Processing then flows to block 356 in which the printed 

It is very common for one component to have multiple circuit board with the solder paste properly applied thereon 

leads on multiple sides. It should be noted images of one (or is provided to a component placement station. The compo- 

more) of the leads from the component may act as the nent placement station can include a so called pick and place 

primary image(s). Images of the rest of the leads from that 60 machine or alternatively, the placement station may involve 

component may act as the target images. For instance, in manual placement of circuit components on the printed 

step 346, image 337 can be treated as the primary image as circuit board. The decision to use automated or manual 

discussed above in conjunction with FIGS. 3-3B. The component placement techniques is made in accordance 

assumption here is that a component should have leads and with a variety of factors including but not limited to the 

joints that look very similar to each other. One or more good 65 complexity of the circuit component, the sensitivity of the 

looking leads and joints should act as the primary images. circuit component to manual or machine handling, technical 

Leads and joints from that component that have defects will limitations of automated systems to handle circuit compo- 
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Dents of particular sizes and shapes and the cost effective- 
ness of using automated versus manual systems. 

Once the circuit component is placed on the printed 
circuit board, processing moves through block 358 in which 
a placement inspection station performs an inspection of the 
placed circuit component. The placement inspection system 
includes an image capturing device to capturing image of 
each of the circuit components of interest. The captured 
image of the circuit component is compared to a target 
image of a placed circuit component using the techniques 
described here and above in conjunction with FIGS. 3-14. In 
response to the instructions from placement inspection sta- 
tion at block 358, processing can return to processing block 
352 or processing block 356 depending upon the results of 
the placement inspection station in block 358. 

Once determination is made in block 358 that the circuit 
component is properly placed and no other defects are 
detected, processing flows to block 360 in which a solder 
reflow station reflows the solder thus coupling the circuit 
component to the printed circuit board. Solder reflow station 
may be provided as an automated station or as a manual 
station. After solder reflow in block 360, processing flows to 
block 362 where a placement and solder joint inspection 
station inspects each circuit component and solder joint of 
interest If not defects are detected, then processing flows to 
block 364 where a populated printed circuit board is pro- 
vided. 

It should be noted that in the above description, the target 
images are all of good examples of paste application, 
component placement, or relflowed solder joints. It is pos- 
sible have a mixed population of both good and bad 
examples in the target images. The results of matching the 
input image to the good and bad examples will tell the 
system whether the image looks more similar to the good 
examples rather than the bad examples (or vice versa). This 
discrimination can be very important in the decision of 3S 
whether to pass or fail the image. Similarity of the primary 
image to a specific bad example can provide information 
regarding the type of defect. Tins information is extremely 
desirable for purposes of tuning the production process. 

Good and bad examples can be collected over time as 
multiple boards are sent through stages 350 to 362 in FIG. 
15. These examples can be used to create or augment the 
target images for each of the three stages. 

Each inspection stage (stages 354, 358 and 362) can 
collect data regarding the number and type of defects found. 
This data can be analyzed and fed back into the system to 
tune the process. For instance at stage 354, the post-paste 
inspection stage may consistently report insufficient paste 
over the whole board. This may mean that not enough paste 
is being applied in step 352. The inspection system may 
consistently report insufficient paste in one area on the 
board, suggesting a clogged opening in the paste stencil. A 
similar analysis may be done for post-placement and post* 
reflow defects. Feedback regarding errors may be sent from 
any inspection station to any part of the process prior to that 
inspection stage. For instance, consistent detection of insuf- 
ficient solder in step 362 (post reflow) may mean that not 
enough paste is being applied in step 352 (pre reflow). 

Having described the preferred embodiments of the 
invention, it will now become apparent to one of ordinary 
skill in the art that other embodiments incorporating their 
concepts may be used. It is felt therefore that these embodi- 
ments should not be limited to disclosed embodiments but 
rather should be limited only by the spirit and scope of the 
appended claims. All publications and references cited 
herein are expressly incorporated herein by reference in their 
entirety. 
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What is claimed is: 

1. An image processing system comprising: 

(a) a search engine; 

(b) an image analyzer coupled to receive images from said 
search engine and to compare a first image in a first 
field of use to a second image in the first field of use 
provided thereto from said search engine; and 

(c) one or more feature modules, each of said one or more 
feature modules coupled to at least one of said search 
engine and said image analyzer with each individual 
one of said one or more feature modules specifying 
information relevant to a particular field of use and 
wherein a particular one of the one or more feature 
modules provides to said image analyzer, application 
specific information which is used by said image ana- 
lyzer to compare the first and second images, wherein 
the information corresponds to at least one of an image 
parameter, an image characteristic and a constraint in 
the first field of use, 

2. The system of claim 1 further comprising an input 
system coupled to one of said search engine and said image 
analyzer, said input system comprising at least one of: a 
graphical user interface; a facsimile system; a camera sys- 
tem; a scanner; a network connection; and a video system. 

3. The system of claim 2 wherein each of said one or more 
feature modules defines at least one measurement to make 
on one or more pixels in an image region neighboring the 
one particular image region. 

4. The system of claim 1 wherein each of said one or more 
feature modules defines at least one particular region of an 
image and at least one particular measurement to make on 
one or more pixels within each of the at least one particular 
image regions. 

5. The system of claim 3 further comprising a storage 
device having at least one image stored therein coupled to 
one of said a search engine, said image analyzer and said 
feature module. 

6. A process for comparing two images comprising the 
steps of: 

(a) providing a compact primary image by providing 
application specific information to collapse at least one 
primary image subregioo; 

(b) aligning a target image having a first predetermined 
characteristic and the compact primary image having 
the first predetermined characteristic; 

(c) dividing the compact primary image into a plurabty of 
image regions each of the image regions of the primary 
image having a set of image region characteristics; 

(d) selecting a primary image region of the compact 
primary image; 

(e) dividing the target image into a plurality of target 
image regions; 

(f) selecting one of the plurality of target image regions; 
and 

(g) comparing one or more image region characteristics of 
the selected one of the primary image region of the 
compact primary image to corresponding image region 
characteristics in the selected one of the plurality of 
target image regions. 

7. The process of claim 6, further comprising the steps of: 
(g) assigning a score indicating the difference between the 

set of image characteristics in the primary image region 
of the compact primary image and the corresponding 
set of image characteristics in the selected one of the 
plurality of target image regions; 
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(h) selecting a next one of the plurality of image regions 
in the target image; 

(i) comparing one or more image region characteristics of 
the selected next one of the primary image region of the 
compact primary image to corresponding image region 5 
characteristics in the selected next one of the plurality 
of target image regions; and 

(j) assigning a score indicating the difference between the 
image region characteristics in the primary image 
region of the compact image and the corresponding 
image region characteristics in the next target image 
region. 

8. The process of claim 7 further comprising the step of 
repeating steps (h)-(j) for each of a predetermined number 
of regions in the target image. 15 

9. The process of claim 8 further comprising the step of 
finding the best match between the primary image region of 
the compact image and one of the predetermined regions in 
the target image. 

10. The process of claim 9, further comprising the steps 

of: 

selecting a next primary image region in the compact 

primary image; and 
repeating steps (h)-Q) for each desired primary image 25 

region. 

U. The process of claim 10, further comprising the steps 

of: 

computing a total match score; and 

providing said total match score as output. 30 

12. The process of claim 6, further comprising the step of: 
combining the set of image region characteristics in first 

ones of the plurality of image regions with the set of 
image region characteristics in second ones of the 
plurality of image regions of the selected image to 35 
provide a plurality of combined image regions which 
represent a compact primary image wherein each of the 
combined image regions are provided having a set of 
image region characteristics which are a combination 
of the image region characteristics of at least a first and 40 
a second image region of the primary image. 

13. A method of manufacturing a printed circuit board 
comprising the steps of: 

(a) performing a manufacturing operation on a printed 
circuit board; 

(b) obtaining an electronic image of the manufacturing 
operation wherein the image includes at least a portion 
of the printed circuit board and wherein the image 
corresponds to a primary image having a first applica- 5Q 
tion specific predeterrnined characteristic; 

(c) aligning the primary image to a target image of the 
manufacturing operation wherein the target image 
includes at least the portion of the printed circuit board 
included in the primary image and wherein the target 55 
image has the first predetermined characteristic; 

(d) dividing the primary image into a plurality of primary 
image regions each having a set of application specific 
primary image region characteristics; 

(e) combining the set of primary image region character- 60 
istics in first ones of the plurality of primary image 
regions with the set of image region characteristics in 
second ones of the plurality of primary image regions 

to provide a plurality of combined image regions which 
represent the primary image wherein each of the com- 65 
bined image regions are provided having a set of image 
region characteristics which are a combination of the 
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image region characteristics of at least first and second 
primary image regions; 

(f) selecting one of the plurality of combined image 
regions; 

(g) dividing the target image into a plurality of target 
image regions; 

(h) selecting one of the plurality of target image regions; 
and 

(i) inspecting the result of the manufacturing operation by 
comparing one or more image region characteristics of 
the selected one of the plurality of combined image 
regions to corresponding image region characteristics 
in the selected one of the plurality of target image 
regions. 

14. The method of claim 13 further comprising the steps 
of: 

capturing an image of a portion of the printed circuit 
board wherein said captured image portion includes an 
image of the printed circuit board having the manufac- 
turing operation applied thereto; 

selecting a region of the captured image; 

aligning a target image and the selected region of the 
captured wherein the target image corresponds to an 
image of a printed circuit board which has been prop- 
erly manufactured in during that particular portion of 
the manufacturing process and wherein each of the 
target and selected images has at least one property; 

dividing the selected image into a plurality of image 
regions each of the images regions having at least one 
property; 

combining properties in predetermined ones of the plu- 
rality of image regions; 

selecting a primary image region within the selected 
image; 

selecting a target image region; 

comparing one or more properties of the primary image 
region within the selected image to a corresponding one 
or more properties in the target image region; and 

in response to the primary image being substantially 
similar to the target image in said comparing step, 
utilizing, as the target images, images of like portions 
of the printed circuit board having the same manufac- 
turing operation applied thereto as the primary image. 

15. The process of claim 11, wherein the step combining 
the set of primary image region characteristics in first ones 
of the plurality of primary image regions with the set of 
image region characteristics in second ones of the plurality 
of primary image regions to provide a plurality of combined 
image regions which compactly represent the primary image 
as a compact primary image further comprises the step of 
combining primary image region characteristics in neigh- 
boring primary image regions to provide combined image 
regions. 

16. The process of claim 13 wherein for each of a 
pretetermined number of regions in the target image the 
method further comprises the step of repeating the steps of: 

(1) selecting a next region in the target image; 

(2) comparing one or more image region characteristics of 
the compact image region to a corresponding one or 
more image region characteristics in the next target 
image region; and 

(3) assigning a score indicating the difference between the 
one or more image region characteristics in the com- 
pact image region and the corresponding one or more 
image region characteristics in the next target image 
region. 
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17. The process of claim 16 further comprising the step of 
finding the best match between the compact image region 
and one of the predetermined regions in the target image. 

18. The process of claim 13, wherein the target image 
corresponds to an image of a printed circuit board portion 5 
which has failed a printed circuit board inspection process. 

19. The process of claim 13 wherein the manufacturing 
operation corresponds to a solder paste application step. 

20. The process of claim 13 wherein said manufacturing 
operation corresponds to a component placement operation. 10 

21. The process of claim 13 wherein said manufacturing 
operation corresponds to a solder re flow operation. 

22. The process of claim 18 wherein said manufacturing 
operation corresponds to a solder operation and the method 
further includes the step of utilizing relative brightness 15 
between predetermined regions at and proximate to a solder 
joint to verify the acceptability of the solder joint. 

23. A memory for storing a model for access by an image 
processing system, the model comprising: 

a plurality of image regions each of the image regions 20 
from a primary image having a set of application 
specific image region characteristics; and 
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a set of image region characteristics wherein the set of 
region characteristics are provided from a combination 
of the set of image region characteristics in first ones of 
the plurality of image regions with second ones of the 
plurality of image regions to provide a plurality of 
combined image regions which represent a compact 
primary image wherein each of the combined image 
regions are provided having a set of image region 
characteristics which are a combination of the image 
region characteristics of at least a first and a second 
image region of the primary image. 

24. The memory of claim 23 farther wherein at least one 
of the combined image regions includes a combined spatial 
and a combined photometric relationship between a first one 
of the plurality of image patches and a second one of the 
plurality of image patches within the model. 

25. The memory of claim 24 wherein the least one of the 
combined image regions includes the characteristics of a 
third one of the plurality of image regions. 

26. The memory of claim 24 wherein each of the plurality 
of image regions correspond to a subset of an entire image. 
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